We present a 3D variational autoencoder (VAE) for encoding microstructure volume elements comprising voxelated orientation data. We show that the model generalises well to unseen textures, grain sizes and aspect ratios. Accurate reconstructions are achieved, and a continuous latent space is constructed. Symmetries in the orientation space are accounted for by mapping to the crystallographic fundamental zone as a preprocessing step, which allows for a continuous loss function to be used. For the loss, we utilise spectral regularisation to enforce continuity in the latent space [1]. The VAE is then used to encode a training set of volume elements that are used as initial configurations in various crystal plasticity (CP) simulations. Microstructural fingerprints extracted from the VAE, which parameterise the volume elements in a relatively low-dimensional latent space, are stored alongside the volume-averaged stress tensor response to applied random average deformation gradient loading paths resulting from CP simulations. This is then used to train a recurrent neural network (RNN), which acts as a surrogate model for the CP simulation. Previous work on RNN-based surrogates for CP have been shown to perform well when correlating between stress and strain [2], but microstructure dependence is often excluded. Suitable fingerprinting has enabled the inclusion of microstructural dependence. We show that, given a statistically equivalent volume element and load path, the trained RNN can accurately predict the resulting stress response. Once trained, the RNN offers significant speed up over running a CP simulation and can ultimately serve as a tool for upscaling volume element responses to a component scale simulation.